Troubleshooting system using device snapshots

ABSTRACT

An information handling system (IHS) troubleshooting system includes a customer IHS including a plurality of components. A customer management system in the customer IHS detects a failure in a managed system and, in response, immediately triggers a device snapshot of the customer IHS. At least one managed system in the customer IHS includes a device snapshot engine that, in response to the customer management system triggering the device snapshot of the customer IHS, immediately create the device snapshot of the customer IHS. A device snapshot storage in the customer IHS stores the device snapshot of the customer IHS. A snapshot communication engine in the customer IHS sends the device snapshot of the customer IHS over the network to a support IHS. The support IHS may load the device snapshot into a virtual IHS and manage the virtual IHS to replicate the failure detected in the managed system for troubleshooting.

BACKGROUND

The present disclosure relates generally to information handling systems(IHSs), and more particularly to system for troubleshooting IHSs usingdevice snapshots.

As the value and use of information continues to increase, individualsand businesses seek additional ways to process and store information.One option is an IHS. An IHS generally processes, compiles, stores,and/or communicates information or data for business, personal, or otherpurposes. Because technology and information handling needs andrequirements may vary between different applications, IHSs may also varyregarding what information is handled, how the information is handled,how much information is processed, stored, or communicated, and howquickly and efficiently the information may be processed, stored, orcommunicated. The variations in IHSs allow for IHSs to be general orconfigured for a specific user or specific use such as financialtransaction processing, airline reservations, enterprise data storage,or global communications. In addition, IHSs may include a variety ofhardware and software components that may be configured to process,store, and communicate information and may include one or more computersystems, data storage systems, and networking systems.

Some IHS users contract with support providers such that when there is aproblem with their IHS, the support provider provides remote support tofix the problem with the IHS. The provision of remote support can raisea number of issues. For some problems, support providers will physicallyreproduce the IHS situation that caused the problem to allowtroubleshooting of that problem in order to solve it. However, some IHSsituations may be very difficult to physically reproduce, and in somecases the problem occurring in the user IHS may never materialize evenwhen the IHS situation is physically reproduced. For example, theproblem of IHSs in a datacenter intermittently moving into an unmanagedstate has been found to not materialize even when the IHS situation wasphysically reproduced. In another example, the problem of scriptfailures for fan modules in IHSs materialized only once in two weeks ofphysically reproducing the IHS situation. In yet another example, theproblem of IHSs reporting certain volumes of storage devices as criticaleven when those volumes were not critical only materialized when the IHSsituation, including over 200 hard disks and over 60 volumes, wasphysically reproduced, an IHS physical reproduction that required twoweeks of time to accomplish. As should be appreciated, such physicallyreproductions are time-consuming and expensive to provide to attempt tosolve a users IHS problem, particularly when the problem does notmaterialize upon the physical reproduction the IHS situation and thuscannot be solved remotely.

Accordingly, it would be desirable to provide an improvedtroubleshooting system.

SUMMARY

According to one embodiment, an information handling system (IHS)troubleshooting system includes a customer IHS that is operable to becoupled to a network, wherein the customer IHS includes: a customermanagement system that is operable to detect a failure in a managedsystem and, in response, immediately trigger a device snapshot of thecustomer IHS; at least one managed system including a device snapshotengine that is operable, in response to the customer management systemtriggering the device snapshot of the customer IHS, to immediatelycreate the device snapshot of the customer IHS; a device snapshotstorage that is operable to store the device snapshot of the customerIHS; and a snapshot communication engine that is operable to send thedevice snapshot of the customer IHS over the network to a support IHS.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view illustrating an embodiment of an IHS.

FIG. 2 is a schematic view illustrating an embodiment of an IHStroubleshooting system.

FIG. 3 is a schematic view illustrating an embodiment of a customer IHSused in the IHS troubleshooting system of FIG. 2.

FIG. 4 is a schematic view illustrating an embodiment of a support IHSused in the IHS troubleshooting system of FIG. 2

FIG. 5 is a flow chart illustrating a method for troubleshooting an IHS.

FIG. 6 is a use case illustrating an embodiment of a portion of themethod of FIG. 5.

FIG. 7 is a schematic view illustrating an embodiment of an IHStroubleshooting system.

FIG. 8 is a schematic view illustrating an embodiment of an IHStroubleshooting system.

FIG. 9 is a use case illustrating an embodiment of a portion of themethod of FIG. 5.

DETAILED DESCRIPTION

For purposes of this disclosure, an IHS may include any instrumentalityor aggregate of instrumentalities operable to compute, classify,process, transmit, receive, retrieve, originate, switch, store, display,manifest, detect, record, reproduce, handle, or utilize any form ofinformation, intelligence, or data for business, scientific, control,entertainment, or other purposes. For example, an IHS may be a personalcomputer, a PDA, a consumer electronic device, a display device ormonitor, a network server or storage device, a switch router or othernetwork communication device, or any other suitable device and may varyin size, shape, performance, functionality, and price. The IHS mayinclude memory, one or more processing resources such as a centralprocessing unit (CPU) or hardware or software control logic. Additionalcomponents of the IHS may include one or more storage devices, one ormore communications ports for communicating with external devices aswell as various input and output (I/O) devices, such as a keyboard, amouse, and a video display. The IHS may also include one or more busesoperable to transmit communications between the various hardwarecomponents.

In one embodiment, IHS 100, FIG. 1, includes a processor 102, which isconnected to a bus 104. Bus 104 serves as a connection between processor102 and other components of IHS 100. An input device 106 is coupled toprocessor 102 to provide input to processor 102. Examples of inputdevices may include keyboards, touchscreens, pointing devices such asmouses, trackballs, and trackpads, and/or a variety of other inputdevices known in the art. Programs and data are stored on a mass storagedevice 108, which is coupled to processor 102. Examples of mass storagedevices may include hard discs, optical disks, magneto-optical discs,solid-state storage devices, and/or a variety other mass storage devicesknown in the art. IHS 100 further includes a display 110, which iscoupled to processor 102 by a video controller 112. A system memory 114is coupled to processor 102 to provide the processor with fast storageto facilitate execution of computer programs by processor 102. Examplesof system memory may include random access memory (RAM) devices such asdynamic RAM (DRAM), synchronous DRAM (SDRAM), solid state memorydevices, and/or a variety of other memory devices known in the art. Inan embodiment, a chassis 116 houses some or all of the components of IHS100. It should be understood that other buses and intermediate circuitscan be deployed between the components described above and processor 102to facilitate interconnection between the components and the processor102.

Referring now to FIG. 2, an IHS troubleshooting system 200 isillustrated. The IHS troubleshooting system 200 includes a network 202that couples a plurality of customer IHSs 204, 206, and 208 to a supportIHS 210. In an embodiment, the customer IHSs 204, 206, and 208 and/orthe support IHS 210 may be the IHS 100, described above with referenceto FIG. 1, and may include some or all of the components of the IHS 100.The network 202 may be a intranet, the Internet, and/or a variety ofother networks known in the art. While three customers IHSs areillustrated coupled a single support IHS, one of skill in the art willrecognize that any number of customer IHSs may be coupled to any numberof support IHSs through the network 202 while remaining within the scopeof the present disclosure.

Referring now to FIG. 3, a customer IHS 300 is illustrated. In anembodiment, the customer IHS 300 may be the IHS 100, discussed abovewith reference to FIG. 1, and/or the customer IHSs 204, 206, and/or 208,discussed above with reference to FIG. 2. The customer IHS 300 includesa management system 302 that is coupled to a plurality of managedsystems 304, 306, and 308. While three managed systems are illustrated,one of skill in the art will recognize that any number of managedsystems will fall within the scope of the present disclosure. Themanagement system 302 is also coupled to a mailbox 310 that includes adevice snapshot storage 310 a and a snapshot communication engine 310 b.The snapshot communication engine 310 b is coupled to the network 202using methods known in the art. In an embodiment, the mailbox 310 is anadministrator mailbox accessible by an administrator of the customer IHS300 through one or more access devices coupled to the customer IHS 300.In an embodiment, the management system 302 is a management server thatincludes a system center operation manager (SCOM) that is operable tomonitor applications and hardware devices. In an embodiment, the SCOMincludes a cross platform data center management server for operatingsystems, hypervisors, and/or other systems known in the art, and mayutilize an interface displaying state, health, and performanceinformation of the managed systems 304, 306, and 308. The SCOM isoperable to provide alerts generated according to some availability,performance, configuration, security situation, and/or other eventidentified in the managed systems 304, 306, and 308.

In an embodiment, the managed systems 304, 306, and 308 may be servers,clients, Modular Disk (MD) arrays, storage systems, chassis managementcontrollers (CMCs), remote access controllers (RACs), printers, and/orvariety of other managed systems known in the art. In an embodiment, theSCOM on the management system 302 is operable to monitor the managedsystems 304, 306, and 308 by pushing agents onto the managed systems304, 306, and 308 and collecting monitoring data from those agents. Inan embodiment, an agent on a managed system may watch one or moresources on that managed system (e.g., an event log such as, for example,a Windows® event log) for specific events or alerts generated by theapplications executing on that managed system. Upon alert occurrence ordetection, the agent is operable to forward the alert to the SCOM, whichmay maintain a database that includes a history of the alerts. The SCOMon the management system 302 may also be operable to execute scripts onthe management server 302 that connect to the managed systems 304, 306,and 308 and pull the monitoring data from the managed systems 304, 306,and 308. For example, the SCOM may monitor storage servers, remoteaccess controllers, printer hardware, and/or a variety of othermonitored systems known in the art, by executing instrumentation scriptson the management system 302. In an embodiment, the management system302 or a device connected to the management system 302 allows monitoringdata to be viewed by a user in various formats (e.g., using the display110, described above with reference to FIG. 1).

In an embodiment, the SCOM is operable to apply filtering rules to thealerts as they are received or retrieved, and those rules may trigger anotification (e.g., an email), generate a network support ticket, and/ortrigger some other workflow known in the art that maybe used incorrecting the cause of the alert. A set of filtering rules specific toa monitored application may be referred to as a management pack. Forexample, a management pack may include instrumentation scripts thatdiscover, classify, and/or create detailed device inventories and/ormonitor device health. Management packs may also monitor software/devicecomponents. In an embodiment, separate management packs for managingservers, laptops, desktops, storage, printers, and/or other networkdevices may be provided. Instrumentation scripts on these managementpacks may then run periodically such that the management system 302 maybuild a complete view of the managed systems 304, 306, and 308 using thecollected information. A troubleshooting management pack, described infurther detail below, is included in the management system 302.Furthermore, some or all of the managed systems 304, 306, and 308 mayinclude management packs, and the management system 302 may includemanagement packs in addition to the troubleshooting management pack. Forexample, the managed systems 304, 306, and 308 may include servermanagement packs, printer management packs, storage management packs,and/or a variety of other management packs known in the art. In anembodiment, management packs on the managed systems 304, 306, and 308may provide one or more device snapshot engines on the managed systemsthat include instructions that are stored on a non-transitory,computer-readable medium and that, when executed by a processor, areoperable to create a device snapshot of the customer IHS, as discussedin further detail below.

Referring now to FIG. 4, a support IHS 400 is illustrated. In anembodiment, the support IHS 400 may be the IHS 100, discussed above withreference to FIG. 1, and/or the support IHS 210, discussed above withreference to FIG. 2. The support IHS 400 includes a customer IHSre-creation engine 402 that includes instructions that are stored on anon-transitory, computer-readable medium (e.g., the system memory 114described above with reference to FIG. 1, the mass storage device 108described above with reference to FIG. 1, and/or other computer-readablemedia known in the art) and that, when executed by a processor, causethe processor to perform a number of operations that are discussed infurther detail below. The customer IHS re-creation engine 402 is coupledto the network 202. As discussed in further detail below, the customerIHS re-creation engine 402 is operable to create a virtual customer IHS404 that may include a plurality of virtual IHSs 404 a, 404 b, and 404 ccorresponding to the managed system 302, 304, and 306. In an embodiment,the customer IHS re-creation engine 402 is also operable to create avirtual management system 406 that is coupled to the virtual customerIHS 404. In another embodiment, the management system 406 may be aphysical IHS that is coupled to the virtual customer IHS 404.

Referring now to FIGS. 3, 4, 5 and 6, a method 500 for troubleshootingan IHS is illustrated. FIG. 6 illustrates a use case 600 that breaks themethod 500 up into functionality that is carried out using managementpacks 602 on the management system 302 and managed systems 304, 306, and308 on the customer IHS 300, an event log 604 (e.g., a Windows® eventlog) on the customer IHS 300, and a troubleshooting management pack 606on the management system 302 in the customer IHS 300, in blocks 502,504, and 506 of the method 500. The method 500 begins at block 502 wherea failure is detected in a managed system in a customer IHS. The SCOM onthe management system 302 in the customer IHS 300 may execute managementpack instrumentation scripts and, in an embodiment, a script may fail orabruptly terminate. For example, the SCOM may execute discovery, a unitmonitor script, and/or other executables known in the art, at block 602a, and at block 602 b the script fails.

The method 500 then proceeds to block 504 where a device snapshot of thecustomer IHS is immediately triggered. In response to the script failureat block 602 b, the management system 302 may detect the script failureand generate a script or executable “failed to run” message in the eventlog 604 at block 604 a. For example, the SCOM may generate a Windows®log event that indicates the context of the failure by including adescription of the failure (e.g., “Script or Executable Failed to Run”),a log of the failure (e.g., “Operations Manager”), an event source(e.g., “Services Module”), and an Event (e.g., “21405/21406”). In anembodiment, the script may have failed due to a variety of reasons knowin the art such as, for example, errors in the managementinstrumentation objects, unexpected values, script errors, etc.

The script or executable “failed to run” message in the event log 604triggers the troubleshooting management pack in the management server302 at block 606 a, and the troubleshooting management pack determinesthe source of the script failure at block 606 b. In an embodiment, thesource of the script failure may be software and/or a driver running ona managed system in the network (that is managed by the SCOM), anothermanagement pack, and/or a variety of other script failure sources knownin the art. The troubleshooting management pack may determine the sourceof a script failure by parsing through events that are logged into anevent log on the managed system 302. Upon determining the source of thescript failure, the troubleshooting management pack generates a devicesnapshot command message in the event log 604 at block 604 b, whichtriggers one or more management packs at block 602 c. In an embodiment,the management pack in the managed system that was determined to be thesource of the script failure at block 606 b is triggered at block 602 c.In another embodiment, a plurality or all of the management packs in themanaged systems 304, 306, and 308 are triggered at block 602 c. Forexample, a device snapshot may be triggered for the software/drivercomponents on the managed systems that report a script failure, andbecause those components are monitored by management packs, all relevantmanagement packs monitoring those software/driver components on themanaged systems are triggered. At block 602 d, the managed system(s)triggered at block 602 c operate to create a device snapshot for thecustomer IHS 300. As discussed above, the management packs providedevice snapshot engines that are operable to create a device snapshotfor customer IHS 300. In an embodiment, block 504 of the method 500 isperformed such that device snapshot engine(s) create the device snapshotof the customer IHS 300 at block 602 d within 5 seconds of the scriptfailure being detected at block 602 b. Thus, the ‘immediate’ triggeringof the device snapshot of the customer IHS 300 at block 504 of themethod 500 causes the device snapshot of the customer IHS 300 to becreated within a few seconds of the detected failure which, inexperimental embodiments, has been found to provide a device snapshot ofthe customer IHS 300 that may be used to replicate the exactconfiguration of the customer IHS 300 at the moment the failureoccurred, discussed in further detail below.

In one example, referring now to FIGS. 3 and 7, a device snapshot may betaken by an IHS troubleshooting system 700. The IHS troubleshootingsystem 700 includes a customer IHS management system 702 coupled to acustomer IHS managed system 704 and a support IHS customer IHSre-creation engine 706. In an embodiment, the customer IHS managementsystem 702 may be the management system 302 in the customer IHS 300,described above with reference to FIG. 3, and the customer IHS managedsystem 704 may be one of the managed systems 304, 306, and 308 in thecustomer IHS 300, described above with reference to FIG. 3. In anembodiment, the troubleshooting system 700 may be used when the managedsystems 304, 306, and 308 in the customer IHS 300 include servers,clients, and/or a variety of other IHSs known in the art. The customerIHS management system 702 includes a server management pack 702 a and atroubleshooting management pack 702 b. The customer IHS managed system704 includes a management instrumentation serializer 704 a (e.g., aWindows® management instrumentation (WMI) serializer), a device managedobject format (MOF) files database 704 b, and a managementinstrumentation error objects database 704 c (e.g., a Windows®management instrumentation (WMI) error objects database). At block 504of the method 500, a management pack in the managed system 704 may causethe management instrumentation serializer 704 a to create the devicesnapshot by capturing the management instrumentation state (e.g., theWindows® management instrumentation state) from the device MOF files inthe database 704 b and/or the management instrumentation error objectsin the database 704 c. For example, the management instrumentationserializer 704 a may use a management instrumentation interface to querya device for all management instrumentation objects on the device thatinclude data pertaining to inventory information and health state of thedevice. The management instrumentation serializer will serialize all theinventory and health data into a device snapshot file that captures thestate of the management instrumentation objects, the health of themanagement instrumentation objects, MOF files in the database, and errorobjects in a standardized formatting language (e.g., Extensible MarkupLanguage (XML)) such that the device snapshop file format may berecognized by both the serializer and a deserializer.

In another example, referring now to FIGS. 3 and 8, a device snapshotmay be taken by an IHS troubleshooting system 800. The IHStroubleshooting system 800 includes a customer IHS management system 802coupled to a customer IHS managed system 804 and a support IHS customerIHS re-creation engine 806. In an embodiment, the customer IHSmanagement system 802 may be the management system 302 in the customerIHS 300, described above with reference to FIG. 3, and the customer IHSmanaged system 804 may be one of the managed systems 304, 306, and 308in the customer IHS 300, described above with reference to FIG. 3. In anembodiment, the troubleshooting system 800 may be used when the managedsystems 304, 306, and 308 in the customer IHS 300 include storageservers, chassis management controllers (CMCs) remote access controllers(RACs), printers, and/or a variety of other systems known in the art.The customer IHS management system 802 includes a management pack 802 aand a troubleshooting management pack 802 b. The customer IHS managedsystem 804 includes a simple network management protocol (SNMP)serializer 804 a coupled to an SNMP walks database 804 b, and a RemoteAccess Controller Administration (RACADM) serializer 804 c coupled to anRACADM command outputs database 804 d. In an embodiment, at block 504 ofthe method 500, a management pack in the managed system 804 may causethe SNMP serializer 804 a to create the device snapshot by generatingSNMP walks in the SNMP walks database 804 b for use in capturing thestate of the managed system 704. The SNMP Serializer uses SNMP WalkRequests to query the device for all SNMP objects on the devicecontaining data pertaining to the inventory information and health statefor the device. The SNMP Serializer will serialize all the inventory andhealth data into a device snapshot file that captures the states of theSNMP objects, the ehealth of the SNMP objects, and error objects in astandardized formatting language (e.g., XML) such that the devicesnapshot file format is recognize by both the serializer and adeserializer. In an embodiment, at block 504 of the method 500, amanagement pack in the managed system 804 may cause the RACADMserializer 804 c to create the device snapshot by saving the state ofall commands in the RACADM command outputs database 804 d for use inreplicating the state of the managed system 704. The RACADM Serializerwill query all the objects supported by the RACADM utility by runningall supported RACADM commands and saving the output of the commands intoa file using a standardized formatting language (e.g., XML.) In anunillustrated example, the device snapshot may be created for ModularDisk (MD) arrays using an Array Bridge Broker (ABB) toolkit insimulation mode to capture and/or simulate the state of the managedsystem. For example, the troubleshooting management pack may run the ABBtoolkit in simulation mode when a failure is encountered in the MDarray, which results in the creation of a device snapshot file where theMD array objects that contain all the inventory and health state datafor the device are saved into a file using a standardized formattinglanguage (e.g., XML). The device may then be replicated by running theABB toolkit in simulation mode, which results in the deserializer modulein the ABB toolkit reading the inventory and failure files, allowing theABB to generate all the data needed by the MD array management pack torecreate the exact customer device state.

The method 500 then proceeds to block 506 where the device snapshot isstored in a device snapshot storage. After creating the device snapshotat block 602 d, the management pack(s) in the managed system(s) generatea “device snapshot created” message in the event log 604 at block 604 c.The “device snapshot created” message in the event log 604 triggers thetroubleshooting management pack in the management server 302 at block606 c, and the troubleshooting management pack sends the device snapshotto a device snapshot storage at block 606 d. In an embodiment, thetroubleshooting management pack sends the device snapshot to the devicesnapshot storage 310 a in the mailbox 310 which, as discussed above, maybe an administrator mailbox controlled by an administrator of thecustomer IHS 300. In another embodiment, a local device snapshotdirectory (e.g., in a managed system 304, 306, and/or 308) may be usedto store the device snapshot before the device snapshot is sent to themailbox 310. In an embodiment, device snapshots stored in the devicesnapshot storage 310 a (or the local device snapshot directory) may bedeleted from the device snapshot storage 310 a after a predeterminedamount of time (e.g., 7 days.)

The method 500 then proceeds to block 508 where the device snapshot issent to a support IHS. The device snapshot communication engine 310 b isoperable to send the device snapshot created at block 504 of the method500 over the network 202 to the support IHS 400. In an embodiment, themailbox 310 may receive an message indicating that the failure wasdetected and that the device snapshot that was created in response tothat detected failure is located in the device snapshot storage 310 a,and the message may include an option to send the device snapshot to thesupport IHS 400 through the network 202 using the device snapshotcommunication engine 310 b. In another embodiment, the device snapshotin the device snapshot storage 310 a may be immediately sent through thenetwork using the device snapshot communication engine 310 b. While anadministrator or other mailbox 310 has been described as storing andsending the device snapshot to the support IHS 400, one of skill in theart will recognize that the device snapshot storage and the devicesnapshot communication engine may be located outside of a mailbox or inanother application without departing from the scope of the presentdisclosure.

Referring now to FIGS. 4, 5, and 7, the method 500 then proceeds toblock 510 where the device snapshot is loaded into a virtual IHS. Atblock 508 of the method, the customer IHS re-creation engine 402 in thesupport IHS 400 receives the device snapshot, and at block 510 of themethod 500 the customer IHS re-creation engine 402 creates a virtualcustomer IHS 404 and loads the received device snapshot of the customerIHS 300 into the virtual customer IHS 404. In an embodiment, the virtualIHSs 404 a, 404 b, and 404 c corresponding to the managed systems 304,306, and 308 are created when the device snapshot is loaded onto thevirtual customer IHS 404. In an embodiment, the loading at block 510 isa relatively simple step that involves creating a virtual machine,installing all the deserializers on the virtual machine (WMI, Net SNMPagent/SNMP virtual MIB module, IIS/RACADM simulator, etc.), and copyingthe device snapshots at specific directories where the deserializers mayread them, which results in a virtual machines that mimics the customersystem.

In one example, the troubleshooting system 700 illustrated in FIG. 7 maybe used to load the device snapshot into the virtual IHS. In thetroubleshooting system 700, the support IHS customer IHS re-creationengine 706 is coupled to a support IHS management system 708 and asupport IHS virtual IHS 710. In an embodiment, the support IHSmanagement system 708 may be the management system 406 in the supportIHS 400 described above with reference to FIG. 4, and the support IHSvirtual IHS 710 may be one of the virtual IHSs 404 a, 404 b, or 404 c inthe virtual customer IHS 404 in the support IHS 400 described above withreference to FIG. 4. As discussed above, the support IHS managementsystem 708 may be a physical management system, or the support IHSmanagement system 708 may be a virtual management system created by thesupport IHS customer IHS re-creation engine 706. The support IHSmanagement system 708 includes a server management pack 708 a. Thesupport IHS virtual IHS 710 is created by the customer IHS re-creationengine 706 and includes a management instrumentation command-line (MIC)710 a (e.g., a Windows® management instrumentation command line (WMIC))coupled to management instrumentation 710 b (e.g., a Windows® managementinstrumentation (WMI)) and virtual management instrumentation 710 c(e.g., a virtual WMI). The MIC 710 a is also coupled to a customerdevice MOF files database 710 d and a management instrumentation errorobjects database 710 e (e.g., Windows® management instrumentation (WMI)error objects). The support IHS virtual IHS 710 may be created byloading the management instrumentation state captured by the managementinstrumentation serializer 704 a in the management system 704 onto thesupport IHS virtual IHS 710. For example, the snapshots captured by theserializer are in the standard format that is understood by thedeserializer. The management instrumentation deserializer will then readthe data files and MOF files containing all the information about thedevice such that the deserializer may recreate the inventory and healthinformation for the device into a format which can be read by themanagement pack in the management system so as to reproduce the failurescenario as seen in the customer system.

In another example, the troubleshooting system 800 illustrated in FIG. 8may be used to load the device snapshot into the virtual IHS. In thetroubleshooting system 800, the support IHS customer IHS re-creationengine 806 is coupled to a support IHS management system 808 and asupport IHS virtual IHS 810. In an embodiment, the support IHSmanagement system 808 may be the management system 406 in the supportIHS 400 described above with reference to FIG. 4, and the support IHSvirtual IHS 810 may be one of the virtual IHSs 404 a, 404 b, or 404 c inthe virtual customer IHS 404 in the support IHS 400 described above withreference to FIG. 4. As discussed above, the support IHS managementsystem 808 may be a physical management system, or the support IHSmanagement system 808 may be a virtual management system created by thesupport IHS customer IHS re-creation engine 806. The support IHSmanagement system 708 may include a printer management pack 808 a, astorage system management pack 808 b, and/or a CMC management pack 808c. The support IHS virtual IHS 810 is created by the customer IHSre-creation engine 806 and includes a net-SNMP agent 810 a that iscoupled to virtual management information base (MIB) modules 810 b andan SNMP walks database 810 c. The support IHS virtual IHS 810 alsoincludes an Internet Information Server (IIS) 810 d coupled to an RACADMsimulator 810 e and a RACADM command outputs database 810 f. In oneembodiment, the support IHS virtual IHS 810 may be created by loadingthe state captured from the SNMP walks generated by the SNMP serializer804 a onto the support IHS virtual IHS 810 (e.g., using the net SNMPagent with the virtual MIB modules 810 b that read off MIB walks). Forexample, the device snapshot captured by the SNMP serializer is in thestandardized format that is understood by the virtual MIB module. TheNet SNMP agent loads the virtual MIB module that will read the devicesnapshot (e.g., SNMP walks saved in an XML format), which allows thevirtual MIB module to recreate the inventory and helath information forthe device into the SNMP objects. Those SNMP objects may then be read bythe SNMP management packs so as to reproduce the failure scenario asseen in the customer system. In another embodiment, a dummy RACADMprotocol may be implemented using the RACADM simulator 810 e and theRACADM command outputs database 810 f to replicate the state of thecustomer managed system using the state of all commands saved by theRACADM serializer 804 c. For example, the device snapshots captured bythe RACADM serializer is in the standardized format understsood by theRACADM simulator. The IIS loads the RACADM simulator that will read thedevice snapshot (e.g., the RACADM command outputs saved in XML format),which allows the RACADm simulator to recreate the inventory and healthinformation for the device into RACADM objects. Those RACADM objets canthem be read by the CMC management packs so as to reproduce the failurescenario as seen in the customer system.

Referring now to FIGS. 4 and 5, the method 500 then proceeds to block512 where the virtual IHS is managed to replicate the failure of thecustomer IHS. The management system 406 in the support IHS 400 is usedto manage the virtual customer IHS 404. Because the virtual customerIHSs 404 is created using a device snapshot taken from the managedsystems 304, 306, and 308 in the customer IHS 300, the virtual IHSs 404a, 404 b, and/or 404 c are substantially identical in operation to themanaged systems 304, 306, and 308, and the management of the virtualcustomer IHS 404 using the management system 406 will replicate thefailure detected at block 502 of the method. The support IHS 400 maythen use the replicated and reproduced customer IHS 300 in the virtualcustomer IHS 404 to troubleshoot the customer IHS 300 in order toanalyze and/or diagnose the customer IHS 300 and determine the cause ofthe failure detected at block 502 of the method 500. The use of thedevice snapshot to create the virtual customer IHS 404 creates a virtualIHS that acts as a real device (i.e., the physical customer IHS 300),replicating the device state and error condition. One of skill in theart will recognize that the creation of the virtual customer IHS 404using the device snapshot allows the state of the customer IHS 300 to bereproduced very quickly (almost immediately) and thus greatlyaccelerates the time necessary to allow troubleshooting of the problemsoccurring in the customer IHS 300. Furthermore, due to the devicesnapshot being taken automatically and immediately upon the detection ofthe failure, troubleshooting the customer IHS 300 is much more likely toallow accurate diagnosis of the cause of the failure relative toconvention means, as the virtual IHS created using the immediately takendevice snapshot will have the same state as the customer IHS 300 thatproduced the failure.

In an embodiment, the management system 302 and/or the managed systems304, 306, and/or 308 in the customer IHS 300 may be operable to capturelogs of the managed systems 304, 306, and/or 308 within a predeterminedtime of the failure (e.g., approximately 5 to 10 minutes prior to andsubsequent to the failure) and send those logs along with any associateddevice snapshot that are created. The support IHS 400 may then use thoselogs to provide context for the failure that occurred in the customerIHS 300 in order to enhance the accuracy and/or time needed totroubleshoot the customer IHS 300.

Referring now to FIGS. 3, 4, 5 and 9, an alternative embodiment of aportion of the method 500 for troubleshooting an IHS is illustrated.FIG. 9 illustrates a use case 900 that breaks the method 500 up intofunctionality that is carried out using management packs 902 on themanagement system 302 and managed systems 304, 306, and 308 on thecustomer IHS 300, an event log 904 (e.g., a Windows® event log) on thecustomer IHS 300, and a troubleshooting management pack 906 on themanagement system 302, in blocks 502, 504, and 506 of the method 500.The method 500 begins at block 502 where a failure is detected in amanaged system in a customer IHS. In an embodiment, the SCOM on themanagement system 302 on the customer IHS 300 executes discovery, a unitmonitor script, and/or other executables known in the art, at block 902a, and at block 902 b an unexpected condition occurs (e.g., a managementinstrumentation collection error such as a Windows® managementinstrumentation (WMI) collection error.)

The method 500 then proceeds to block 504 where a device snapshot of thecustomer IHS is immediately triggered. In response to the unexpectedcondition at block 902 b, the management pack(s) in the managedsystem(s) create a device snapshot for the customer IHS 300. Asdiscussed above, the management packs provide device snapshot enginesthat create a device snapshot for the customer IHS 300. In anembodiment, block 504 of the method 500 is performed such that devicesnapshot engine(s) create a device snapshot of the customer IHS 300 atblock 902 c within 5 seconds of the unexpected condition being detectedat block 602 b. Thus, the ‘immediate’ triggering of the device snapshotof the customer IHS 300 at block 504 of the method 500 causes devicesnapshot of the customer IHS 300 to be created within a few seconds ofthe detected failure which, in experimental embodiments, has been foundto provide a device snapshot of the customer IHS 300 that provides theexact configuration of the customer IHS 300 at the moment the failureoccurred.

The method 500 then proceeds to block 506 where the device snapshot isstored in a device snapshot storage. After creating the device snapshotat block 902 c, the management pack(s) in the managed system(s) generatea “device snapshot created” message in the event log 904 at block 904 a.The “device snapshot created” message in the event log 904 triggers thetroubleshooting management pack in the management server 302 at block906 a, and the troubleshooting management pack sends the device snapshotto a device snapshot storage at block 906 b. The method 500 thenproceeds at described above with reference to FIG. 5.

In another embodiment, device snapshots may be used for reproducingenterprise scale (e.g., very large and complex) customer IHSconfigurations (e.g., a customer IHS including servers with 100 storagedevice (or virtual devices)). In such an embodiment, when a user invokesa task on the managed system (e.g., 304, 306, and/or 308) in thecustomer IHS 400, the invoking of that task triggers the managementpack(s) on the managed system(s) to create a device snapshot of thecustomer IHS 300. That device snapshot may then be transferred to thedevice snapshot storage 310 a and sent to the support IHS 400 at anytime to be loaded on a virtual IHS and used to troubleshoot problemswith the customer IHS.

Thus, a system and method for troubleshooting an IHS are provided that,in response to detecting a failure in a managed system, automaticallyand immediately create a device snapshot of a customer IHS for use witha support IHS to create a virtual IHS that is substantially identical tothe customer IHS in order to allow troubleshooting the customer IHS suchthat the cause of the failure may be diagnosed. The virtual IHS may becreated very quickly relative to convention methods (e.g., compared tothe use of logs that require determinations of the state of the device)and provide a virtual IHS in a state that is substantially identical tothe customer IHS immediately following the failure, greatly increasingIHS troubleshooting time and accuracy. Furthermore, the device snapshotsare created automatically, eliminating the need for an administrator ofthe customer IHS to reproduce the failure or unexpected condition, andthe administrator may need only forward the device snapshot to a supportIHS in order to have the failure diagnosed.

Although illustrative embodiments have been shown and described, a widerange of modification, change and substitution is contemplated in theforegoing disclosure and in some instances, some features of theembodiments may be employed without a corresponding use of otherfeatures. Accordingly, it is appropriate that the appended claims beconstrued broadly and in a manner consistent with the scope of theembodiments disclosed herein.

What is claimed is:
 1. An information handling system (IHS)troubleshooting system, comprising: a customer IHS that is configured tobe coupled to a network, wherein the customer IHS includes: a customermanagement system that is configured to detect a failure in a managedsystem and, in response, immediately trigger a device snapshot of thecustomer IHS, wherein the customer management system includes: a firstmanagement pack that detects the failure and, in response, generates afirst log event that indicates the failure; and a second management packthat detects the first log event indicating the failure and, inresponse, generates a second log event to trigger the device snapshot;at least one managed system including a device snapshot engine that isconfigured, in response to the customer management system triggering thedevice snapshot of the customer IHS, to immediately create the devicesnapshot of the customer IHS; a device snapshot storage that isconfigured to store the device snapshot of the customer IHS; and asnapshot communication engine that is configured to send the devicesnapshot of the customer IHS over the network to a support IHS.
 2. Thesystem of claim 1, wherein the device snapshot engine is configured tocreate the device snapshot of the customer IHS within 5 seconds of thecustomer management system detecting the failure in the managed system.3. The system of claim 1, further comprising: an electronic mailbox thatincludes both the device snapshot storage and the snapshot communicationengine.
 4. The system of claim 1, wherein the failure includes a scriptfailure.
 5. The system of claim 1, further comprising: wherein thecustomer management system is configured to capture logs of the managedsystem before and after the detected failure and send the logs of themanaged system over the network to the support IHS.
 6. The system ofclaim 1, wherein the managed system includes: a third management packthat detects the second log event triggering the device snapshot and, inresponse, creates the device snapshot and generates a third log eventindicating that the device snapshot was created; wherein the secondmanagement pack detects the third log event indicating that the devicesnapshot was created and, in response, sends the device snapshot to thedevice snapshot storage.
 7. The system of claim 1, further comprising: asupport IHS that is configured to be coupled to the customer IHS throughthe network, wherein the support IHS includes: a customer IHSre-creation engine that is configured to receive the device snapshot ofthe customer IHS over the network from the snapshot communication engineand load the device snapshot of the customer IHS into a virtual IHS; anda support IHS management system that is configured manage the virtualIHS such that the failure in the managed system that was detected by thecustomer management system is replicated for troubleshooting.
 8. Aninformation handling system (IHS) troubleshooting system, comprising: acustomer IHS that is configured to be coupled to a network, wherein thecustomer IHS includes: at least one managed system including a devicesnapshot engine that is configured, in response to detecting anunexpected condition, to immediately create a device snapshot of thecustomer IHS, wherein the at least one managed system includes: a firstmanagement pack that detects the unexpected condition and, in response,creates the device snapshot and generates a first log event thatindicates the device snapshot was created; a customer management systemthat is in response to determining the device snapshot of the customerIHS has been created, to send the device snapshot to a device snapshotstorage, wherein the customer management system includes: a secondmanagement pack that detects the first log event indicating that thedevice snapshot was created and, in response, sends the device snapshotto the device snapshot storage; and a snapshot communication engine thatis configured to send the device snapshot of the customer IHS over thenetwork to a support IHS.
 9. The system of claim 8, wherein the devicesnapshot engine is configured create the device snapshot of the customerIHS within 5 seconds of detecting the unexpected condition.
 10. Thesystem of claim 8, further comprising: an electronic mailbox thatincludes both the device snapshot storage and the snapshot communicationengine.
 11. The system of claim 8, wherein the unexpected condition isdetected in response to the execution of a script.
 12. The system ofclaim 8, further comprising: wherein the customer management system isconfigured to capture logs of the managed system before and after theunexpected error and send the logs of the managed system over thenetwork to the support IHS.
 13. The system of claim 8, furthercomprising: a support IHS that is configured to be coupled to thecustomer IHS through the network, wherein the support IHS includes: acustomer IHS re-creation engine that is configured to receive the devicesnapshot of the customer IHS over the network from the snapshotcommunication engine and load the device snapshot of the customer IHSinto a virtual IHS; and a support IHS management system that isconfigured to manage the virtual IHS such that the failure in themanaged system that was detected by the customer management system isreplicated for troubleshooting.
 14. A method for troubleshooting aninformation handling system (IHS), comprising: detecting, using a firstmanagement pack in a customer management system, a failure in a managedsystem of a customer IHS and, in response, generating, using the firstmanagement pack, a first log event that indicates the failure; anddetecting, using a second management pack in the customer managementsystem, the first log event indicating the failure and, in response,generating, using the second management pack, a second log event totrigger a device snapshot; in response to generating the second logevent to trigger the device snapshot, immediately triggering a devicesnapshot of the customer IHS; storing the device snapshot of thecustomer IHS in a device snapshot storage; and sending the devicesnapshot of the customer IHS over a network to a support IHS.
 15. Themethod of claim 14, further comprising: receiving the device snapshot ofthe customer IHS over the network by the support IHS and loading thedevice snapshot of the customer IHS into a virtual IHS; and managing thevirtual IHS such that the failure in the managed system that wasdetected is replicated for troubleshooting.
 16. The method of claim 14,wherein the device snapshot of the customer IHS is taken within 5seconds of detecting the failure in the managed system.
 17. The methodof claim 14, further comprising: detecting, using a third managementpack in the managed system, the second log event triggering the devicesnapshot and, in response, creating, using the third management pack,the device snapshot and generating, using the third management pack, athird log event indicating that the device snapshot was created;detecting, using the second management pack, the third log event and, inresponse, sending, using the second management pack, the device snapshotto the device snapshot storage.
 18. A method of claim 14, fortroubleshooting an information handling system (IHS), comprising:detecting, using a first management pack in a managed system, anunexpected condition and, in response, creating, using the firstmanagement pack, a device snapshot of a customer IHS and generating,using the first management pack, a first log event that the devicesnapshot was created; detecting, using a second management pack in acustomer management system, the first log event indicating that thedevice snapshot was created and, in response, sending, using the secondmanagement pack, the device snapshot to the device snapshot storage; andsending the device snapshot over a network to a support IHS.